How we cut median build time from 4m 12s to 47s — without buying a single faster machine.

On the morning of November 7, 2025, our median build time crossed five minutes. Not the 99th percentile — the median. The graphs were leaning the wrong way for three quarters in a row, and we'd run out of "it's because we're growing" as an excuse.

This is the story of how we got it down to 47 seconds — across a fleet of 14,000 builds a day, on the same hardware we had in 2024. Six pull requests, three dead ends, and one Python script that we were genuinely embarrassed by.

01The numbers, before any of this

Here's where we started. Same chart, two lines, six months apart. The "before" line was reactive — we'd built the pipeline three years ago and most of it had been left alone since.

If you skim only one chart, that's it. Three step changes, three pull requests, three engineers — Pavel (me), Lia, and a very patient intern named Aki. Below is the actual order it happened and what we learned, including the parts that didn't work.

02Why builds were actually slow

I spent two weeks just looking. No code changes. The temptation when something is slow is to immediately start writing patches, but for a system that ran 14,000 builds a day across 18 regions, I needed to know which build I was looking at.

I sampled 80 random builds across the week — 40 from EU regions, 40 from US — and walked through them span-by-span in our trace UI. The results were not what I expected.

The three things eating most of the time

• Layer pulls from the cache registry were sequential. Even when a build had 12 layers cached, we pulled them one by one. On a 220 MB image, that's 18–24 seconds of pull time when the parallel theoretical minimum was 3–4.
• The cache key was wrong on roughly a quarter of cached layers. We were hashing the wrong file (a config file that changed on every commit, instead of the lockfile). So things we'd carefully cached were re-built every commit.
• Pushing the final image after the build was sequential too, and we were pushing one layer at a time to one region. Then another region. Then another. For 18 regions, this added 90+ seconds to every build.

None of these is a clever optimization. They were just the kind of thing that creeps in when nobody owns a system end-to-end for a year and a half.

03PR #1 — Cache layer dedup

The first change was the cache-key fix. We were hashing build.config.json, which was generated from environment variables at build time and changed on every CI run. We needed to be hashing the lockfile.

The fix was small. The diff was small. The effect was not small:

1 func cacheKey(step *BuildStep) string {
2   h := sha256.New()
3   h.Write(step.Command)
4 - h.Write(readFile("build.config.json")) // changes per run
5 + h.Write(readFile(step.LockfilePath))    // stable per release
6   return hex.EncodeToString(h.Sum(nil))
7 }

Cache hit rate moved from 39% to 82% in 36 hours. Median build time dropped from 4m 12s to 2m 18s — about half of the eventual win, in a single change.

04PR #3 — Parallel push, region-aware

The next big win was pushing the final image. We had a builder that, after producing an image, would push it region-by-region: eu-central first, then eu-west, then us-east, and so on. Each push was 5–8 seconds, and there were 18 of them.

The "obvious" fix was to push to all 18 in parallel. But our registry has per-region rate limits, and we'd hit them hard if we did that naively. The actual fix had two parts:

• Push within a region in parallel — the registry handles concurrent PUTs for different layer digests fine, and the network bottleneck is the upstream pipe to the registry, not the registry itself.
• Push to regions in waves of 3, with each wave starting before the previous one finished. This kept us under the rate limit and used all available bandwidth.

The pull-side speed-up came along for free from the same code path — we already had the parallel-fetch primitive working in the push direction.

05The Python script we were embarrassed by

This is the part that took longest to ship, partly because we kept rewriting it, and partly because nobody wanted to look at the existing version.

Somewhere in 2023, a previous engineer wrote a Python script called finalise.py that ran after every build. Its job was to update a metadata record in our control database. It ran serially, took 7–14 seconds, and was responsible for about 4% of total build time.

It was 380 lines of Python making 34 separate HTTP requests, in a loop, with no connection pooling. It also re-read a 4 MB JSON manifest from disk on each iteration.

We rewrote it in Go, in 90 lines, using one connection and one in-memory copy of the manifest. The new version takes 0.6 seconds on the slow path. That's not even the interesting story — the interesting story is that this had been on three different engineers' "I'll look at it next sprint" lists for over a year. It was the perfect size of problem to keep getting deferred.

06Three things that didn't work

I want to be honest about this: we tried things that didn't work. Some of them looked great in a small test and got worse in production.

Dead end 1 · Pre-warming the cache from the previous build

The idea: when a build starts, immediately start pulling layers from the previous build of the same service. If they're shared (they usually are), they'll be hot when we need them. In benchmarks: 8% improvement. In production: 1% worse, because the pre-warm contended with the actual pulls and saturated our edge bandwidth.

Dead end 2 · "Smart" build ordering

The idea: schedule fast services first so the queue cleared faster. In benchmarks: looked promising. In production: caused two outages of our build scheduler within a week, both because the "smart" code had a corner case where it decided every service was fast and ran 400 in parallel.

Dead end 3 · Bazel

I'm not going to litigate Bazel here. We tried it for two months. The team that tried it didn't ship anything else for two months. We rolled back to docker buildx with our own cache layer, and shipped four PRs in three weeks.

07What we'd do next

Where to next? A few things on my list, in rough priority order:

• Get p99 below 90 seconds. Median is great, but the tail is still 2m+ on cold builds. The wins there are different — most are about pre-built base layers and BuildKit's --cache-to remote backend.
• Per-org cache pools. Right now, every org shares our global layer cache. That's efficient but it means a busy org can evict another org's cached layers. Sharded pools, sized by paid tier, should solve it.
• A real benchmark suite, running on production hardware with production-shaped load. We have one, but it's three years old and tests a workload nobody runs anymore.

That's mostly it. Six pull requests, no new hardware, three engineers, and one Python script we still occasionally apologize for. Email pavel@scalable.systems if you want to dig into any of it.